Thunderbolt Display - Waking up issue

I use a 27" iMac (El Capitan) and I connect two Thunderbolt Displays, and I stay updated with all updates. The issue I run into is when I wake my computer up that one of my Thunderbolt displays sometimes does not turn on which forces me to unplug both Thunderbolt displays and re-connect them and then both displays fire up and ready to go. But this is annoying and I should not have to do this. Both of Thunderbolt displays are connected directly to the iMac. Does anyone else have this problem or better yet have a solution?